Sound effects have long been used in video game and other computer-generated sound applications. A variety of techniques are known for generating sound effects by computer. One common technique involves using a microphone and recording equipment to record actual sounds from the real world, and storing such sounds in a wave table for real-time read out and reproduction during video game play. Sounds can also be recorded in a wave table through use of a musical synthesizer. The resulting sound effects can be very realistic—adding realism and enjoyment to the entertainment experience.
In some circumstances, it may be desirable to allow the video game player to in some sense control the sound effect. For example, in a driving game, it is desirable for an engine noise sound effect to respond to user manipulation of a “throttle” control (which, for example, may be an actual pedal in an arcade game, but in a home video game context is typically a joystick or other hand-held control). Similarly, in a boat racing game, realism can be increased by changing the sound corresponding to the rate at which waves lap the fiberglass hull of a virtual boat in response to the speed at which the virtual boat is traveling as well as its direction. In other contexts, it may be desirable to change a sound effect depending on what is going on in the game. For example, as the intensity of a virtual rain storm increases, the sound of falling rain should change from a light patter to a driving roar.
One way to accomplish such effects is to provide multiple wave tables storing different variations (i.e., different waveforms) on the same sound effect, and to cross fade from one wave table to another. Thus, for example, as the intensity of a rain storm increases, it is possible to cross fade from one sound effect wave table storing the sound of light rain falling to another sound effect wave table storing the sound of driving rain. In other words, this technique involves storage and playing of a single waveform that represents a real-world sound, or choosing between various waveforms that represent the different types of related sounds. For example, one waveform may represent the sound of a car hitting a wall at 10 mph and another waveform may represent the sound of the car hitting a wall at 100 mph. The appropriate waveform is then selected based on the event for which a sound effect is desired. While this technique has successfully been used to create sound effects in video games and other applications, further improvements in the methodology of sound generation for such applications is desired in order to further increase the realism thereof in a dynamic and exciting manner.
We have now developed and implemented an improved method and system for dynamically generating realistic sound effects in interactive multimedia applications, such as, but not limited to, videogame applications. The improved method and system involves reading out a number of different sound effect wave tables substantially simultaneously, and mixing them together with parameterized volume and pitch read out transfer function characteristics to provide a mixed, complex sound based on a number of wave table outputs. The resulting real-time interactive sound modeling technique allows us to use a continuously-variable user interactivity parameter (e.g., such as that produced by an analog joystick or other interactive control) to provide excellent real-time sonic feedback to the user.
In accordance with the invention, a single “sound,” such as a car hitting a wall, is represented as multiple waveforms whose pitches and volumes are adjusted by one or more parameters. The adjusted waveforms are then mixed together to create a final sound that is output through an audio system. The number of parameters used and what they each represent can vary depending on the particular application in which the invention is implemented. More particularly, the number of different waveforms used and the number of different parameters used to adjust the characteristics of each of the waveforms, as well as what each parameter represents, is determined by the nature of the sonic event being modeled, thereby allowing a high degree of realism and complexity regardless of the particular application in which the invention is employed.
In accordance with the invention, one or more sound models are defined and stored for the sonic event(s) (or sonic environment(s)) that can occur in the multimedia application (e.g., videogame). The sound model includes all information describing how the one or more sounds that are part of the sound model are adjusted by the one or more parameters that are defined in the model. Thus, the sound model includes one or more waveforms, one or more parameters, and, for each waveform and each parameter, a corresponding pitch and volume adjustment function (or graph) referred to herein as an envelope. Thus, in the preferred embodiment, for each waveform and parameter in a sound model of the invention, there is a pitch adjustment function (or envelope) and a volume adjustment function (or envelope). These envelopes define how the pitch and volume of each waveform in the model are adjusted by the current value of each parameter in the model to determine the appropriate adjusted waveform at any particular time during the application.
The parameters may be any suitable parameter, such as the speed at which a player controlled object is moving, the angle at which a player controller object hits another object, the power that the player is giving to a player controller object, etc. The parameters are preferable defined such that they relate (either directly or indirectly) to an action that the player has taken in the game or a condition that the player has caused in the game. For example, one parameter may be the speed at which the player is driving a virtual vehicle, while another parameter may the angle at which the player has caused his virtual vehicle to hit another virtual vehicle or obstacle. A single parameter or multiple parameters may be used in accordance with the invention.
In addition, the waveforms in the sound model are preferably selected to represent various sounds that can occur as result or a real-world experience being represented in the multimedia application. For example, in a driving game, one waveform could represent the sound of the vehicle engine, another waveform could represent the sound of the exhaust pipes, another waveform could represent the sound of the wind, and still another waveform could represent the sound of the tires contacting the road surface. In accordance with the invention, each of the waveforms in the sound model have their pitch and volume (and possible other sound characteristics, such as, for example, attack and decay) parameterized using the particular parameters selected for the sound model. The sound model is then used to dynamically generate real-time final sound for the application, by determining current values of the parameters based on the current application condition (e.g., the current gameplay condition), adjusting the volume and pitch of the waveforms based on current value of the parameters, and mixing the adjusted waveforms together to obtain the final sound.
In accordance with the invention, when multiple waveforms having overlapping pitches and volumes are used in the sound model, together with multiple parameters that adjust those pitches and volumes, very complex and realistic sounds are created, thereby enhancing the gameplay experience or other type of interactive experience in which the invention is implemented.